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V- V*  V  V 


ARI  Research  Reports  and  Technical  Reports  are  intended  for  sponsors  of 
R&D  tasks  and  for  other  research  and  military  agencies.  Any  findings  ready 
for  implementation  at  the  time  of  publication  are  presented  in  the  last  part 
of  the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recom¬ 
mendations  for  official  action  normally  are  conveyed  to  appropriate  military 
agencies  by  briefing  or  Disposition  Form. 


FOREWORD 


The  Curriculum  and  Evaluation  Team  of  the  Instructional  Technology  Sys¬ 
tems  Technical  Area  of  the  U.S.  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences  performs  evaluation  and  curriculum  development  applicable 
to  military  education  and  training.  A  major  focus  of  this  research  is  the 
development  of  information  on  which  the  Department  of  the  Army  can  base  de¬ 
cisions  about  its  Basic  Skills  Education  Program. 

This  report  summarizes  three  years  of  evaluation  research  on  the  Army's 
Basic  Skills  Education  Program.  General  findings  are  that  the  program  im¬ 
proves  the  skills  in  which  eligible  soldiers  are  deficient.  However,  many 
soldiers  graduate  from  the  program  without  achieving  criterion  scores  on 
standard  tests.  Common  factors  that  may  dilute  program  effectiveness  include 
teachers'  lack  of  specialized  training  and  experience,  the  wide  range  of  skill 
levels  within  classes,  and  ongoing  personnel  turbulence. 

This  research  effort  was  supported  by  the  Office  of  the  Adjutant  General 
of  the  Army  and  the  Training  and  Doctrine  Command. 


EDGAR  M.  JOHNSON 
Technical  Director 
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UfRCE  YEARS  OF  EVALUATION  OF  THE  ARMY’S  BASIC  SKILLS  EDUCATION  PROOR^-.M 

EXECUTIVE  SUMMARY 


Requirement : 

The  Department  of  the  Army  has  a  need  for  information  on  which  to  base 
decisions  about  the  future  of  its  Basic  Skills  Education  Program. 


Procedures : 

Two  major  sources  were  used  to  compile  information  about  standard,  pilot, 
revised,  and  developing  basic  skills  programs: 

1.  Data  files  from  the  Training  and  Doctrine  Command,  Enlisted  Master 
File,  and  Defense  Manpower  Document  Center,  and 

2.  Field  visits  to  Army  posts  in  the  continental  United  States,  Germany, 
and  Panama.  During  these  visits,  program  activities  were  observed  and  admini¬ 
strators,  teachers,  participants,  graduates,  noncommissioned  officers,  and 
Commanders  were  given  questionnaires  and  were  interviewed. 


Findings; 

All  programs  improve  soldiers'  basic  skills.  However,  many  soldiers 
graduate  from  basic  skills  progreims  without  achieving  criterion  scores  on 
standard  tests.  Soldiers  show  greater  enthusiasm  for  job-related  curricula 
than  for  materials  lacking  military  content,  and  express  positive  opinions 
about  teachers.  Factors  common  to  all  programs  that  may  influence  their 
effectiveness  are  teachers'  lack  of  specialized  training  and  experience,  the 
wide  range  of  skill  levels  within  classes,  and  reduced  resources  and  personnel 
turbulence  resulting  from  the  lov.’-bid  system. 


Utilization  of  Findings: 

The  Department  of  the  Army  can  use  these  findings  to  guide  future 
decisions  concerning  the  Basic  Skills  Education  Program. 


THREE  YEARS  OF  EVALUATION  OF  THE  ARMY'S  BASIC  SKILLS  EDUCATION  PROGRAM 
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THREE  YEARS  OF  EVALUATION  OF  THE  ARMY'S  BASIC  SKILLS  EDUCATION  PROGRAM 


In  1980,  the  Army  Research  Institute  was  tasked  by  the  Office  of  the 
Adjutant  General  of  the  Army  to  carry  out  an  evaluation  of  the  Army's  Basic 
Skills  Education  Program.  Under  contract  with  the  Army  Institutes  for  Re¬ 
search,  this  effort  has  been  ongoing  since  February  1981.  During  this  period, 
a  great  number  of  reports  and  memorandums  documenting  evaluation  activities 
and  findings  have  been  produced.  The  following  report  has  been  prepared  to 
consolidate  and  summarize  this  information.  Detailed  discussions  of  evalu¬ 
ation  results  and  data  displays  can  be  found  in  the  reports  cited  in  the  text. 


GENERAL 

The  Basic  Skills  Education  Program  (BSEP)  is  a  part  of  the  Army  Continu¬ 
ing  Education  System — the  primary  delivery  system  for  training  and  educational 
opportunities  designed  to  help  soldiers  succeed  in  their  careers.  Hie  programs 
grouped  under  the  BSEP  rubric  provide  enabling  skills  in  language,  literacy, 
arithmetic,  computation,  and  speaking.  The  underlying  rationale  for  BSEP  is 
that  some  enlistees  lack  sufficient  literacy  and/or  English-language  skills 
to  perform  their  jobs  effectively,  and  that  these  deficiencies  can  be  remedied 
by  BSEP  programs. 

BSEP  I  designates  programs  in  literacy  and  English  as  a  second  language 
(ESL)  offered  at  the  training  base  prior  to  or  concurrent  with  Initial  Entry 
Training  (lET).  BSEP  II  programs  involve  soldiers  assigned  to  permanent 
duty  stations  after  completing  lET.  All  of  the  programs  are  monitored  by 
Education  Services  Officers  (ESO)  and  instruction  is  delivered  by  contract 
teachers.  During  the  period  covered  by  this  report,  teachers  had  either 
individual  contracts  or  were  employed  by  institutions  contracted  to  provide 
instructional  services.  Each  contract  institution  supplied  its  own  syllabus, 
curriculum  materials,  lesson  plans,  and  instructional  approach  to  literacy 
training.  In  the  case  of  ESL  programs  in  both  BSEP  I  and  BSEP  II,  most 
contractors  used  the  American  Language  Course  developed  by  the  Defense 
Language  Institute  (DLI).  However,  adaptations  of  the  course  materials  to 
the  needs  of  differing  groups  of  soldiers,  and  the  manner  in  which  teachers 
tended  to  enhance  classroom  instruction  by  introducing  materials  such  as 
military  charts  and  models,  resulted  in  substantial  instructional  differences 
from  class  to  class.  It  is  important  to  note,  however,  that,  although  BSEP 
delivery  methods  varied,  program  goals  remained  constant;  "...to  satisfy  the 
skill  development  and  occupational  needs  of  the  Army  in  defense  of  the 
nation;  and  increase  soldiers'  career  potential,  job  satisfaction,  and 
educational  growth"  (Array  Regulation  621-5,  1981). 

Potential  candidates  for  BSEP  training  are  soldiers  who  fail  to  meet 
the  criterion  score  on  the  General  Technical  (GT)  subtest  of  the  Armed 
Services  Vocational  Aptitude  Battery  (ASVAB)  or  who  speak  English  as  a 
second  language.  These  soldiers  are  screened  for  enrollment  eligibility 
on  the  basis  of  scores  on  the  Test  of  Adult  Basic  Education  (TABE)  for 
general  literacy  training  and  the  English  Comprehension  Level  Test  (ECLT) 
for  language  training.  Those  who  fail  to  meet  test  criteria  are,  at  the 
discretion  of  their  Commanders,  permitted  to  enroll  in  the  program. 
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Another  way  to  participate  in  BSBP  is  by  means  of  command  referral. 

That  means  that  soldiers'  supervisors  judge  that  they  need  literacy  or 
English-language  training  in  order  to  succeed  in  lET  or  in  their  duty  assign¬ 
ments.  A  third  way  to  enroll  in  BSEP  is  by  means  of  individual  requests 
made  by  soldiers  and  approved  by  their  commanders. 

Not  all  eligible  soldiers  enroll  for  BSEP  training.  In  particular. 
Enlisted  Reserve  and  National  Guard  eligibles  commonly  find  it  difficult  to 
extend  their  annual  training  requirements  to  accommodate  BSEP.  Even  in  the 
case  of  Active  Army  eligibles,  participation  is  disapproved  when  soldiers 
cannot  be  spared  from  their  jobs.  As  a  result,  it  is  possible  to  compare, 
on  a  variety  of  indices,  performance  of  BSEP  graduates  with  that  of  soldiers 
eligible  for  BSEP  who  did  not  enroll. 

In  addition  to  evaluating  established  literacy  and  ESL  programs,  our 
task  encompassed  pilot  and  revised  programs,  as  well  as  programs  under  de¬ 
velopment.  He  also  carried  out  a  Needs  Assessment  for  a  program  currently 
being  developed,  and  evaluated  materials  developed  to  identify  prerequisite 
competencies  needed  by  soldiers  for  satisfactory  performance  of  job  tasks. 

All  of  these  activities  are  described  below. 

The  first  group  of  programs  to  be  discussed,  BSEP  I  programs,  include 
the  BSEP  I  literacy  programs,  two  special  programs  developed  for  specific 
Military  Occupational  Specialties  (MOS)  and  materials  developed  to  provide  a 
basis  for  production  of  job-relevant  BSEP  curricula  covering  94  of  the  Army's 
highest  density  MOS. 


BSEP  I  PROGRAMS 

Basic  Literacy  Programs 

These  programs,  offered  at  training  bases,  provide  soldiers  whose  screen¬ 
ing  tests  result  in  reading  levels  below  fifth  grade  with  training  in  remedial 
reading,  writing,  and  computing  skills.  The  training  usually  lasts  from  3  to 
6  weeks,  and  each  Army  post  selects  a  competitive  contractor  who  provides 
instructional  materials  and  teachers  (Krug,  Hahn,  &  Wise,  1983). 

Our  method  for  evaluating  these  programs  was  to  analyze  data  files  from 
the  Training  and  Doctrine  Command  (TRADOC),  Enlisted  Master  File,  and  the 
Defense  Manpower  Document  Center.  These  files  covered  records  of  both  pro¬ 
gram  participants  and  eligible  nonparticipants  for  the  years  1979,  1980,  and 
1981.  Variables  analyzed  included  test  scores  (Adult  Basic  Learning  Examina¬ 
tion,  TABE,  and  ASVAB),  rates  of  attrition,  pay  grades.  Enlisted  Efficiency 
Report  (EER)  scores,  and  Skill  Qualification  Test  (SQT)  scores  (Krug  &  Wise, 
1982a) . 

When  participants  and  eligible  nonparticipants  were  matched  for  ability 
level  (screening  test  scores)  and  length  of  service,  the  average  pay  grade 
of  participants  was  higher  than  that  of  eligible  nonparticipants.  In  addition, 
attrition  rates  were  slightly  lower  for  participants  than  for  nonparticipants. 
No  significant  differences  were  found  between  these  groups  for  EER  or  SQT 
scores.  In  terms  of  scores  resulting  from  testing  prior  to  and  following 


training,  BSEP  was  found  to  result  in  modest  gains  in  grade-level  scores  in 
reading,  vocabulary,  spelling,  arithmetic  computation,  and  problem  solving. 


Although  overall  attrition  was  lower  for  BSEP  graduates,  this  advantage 
was  demonstrated  by  soldiers  in  units.  No  attrition  advantage  showed  up  in 
lET  either  in  Basic  Combat  Training  (BCT)  or  in  Advanced  Individual  Training 
(AIT) .  This  may  indicate  that  BSEP  has  a  greater  impact  on  unit  demands 
than  on  those  of  lET.  The  data  showed,  however,  that  BSEP  exit  test  scores 
positively  correlated  with  success  in  AIT,  i.e.,  twice  as  many  low-scoring 
soldiers  attrited  as  did  high-scoring  soldiers  (Krug,  Hahn  &  Wise,  1983). 


MOS  Baseline  Sltills  Programs 

One  of  the  two  projects  entitled  MOS  Baseline  Sicills  called  for  pro¬ 
duction  of  materials  that  would  permit  the  development  of  curricula  to  teach 
soldiers  competencies  needed  to  perform  94  of  the  Army's  highest  density  MOS, 
as  well  as  common  soldier  tas)cs  (CST).  This  required  TRADOC's  contractor  to 
analyze  MOS  and  CST,  identify  relevant  prerequisite  competencies,  cluster 
the  competencies  common  to  two  or  more  MOS  to  reduce  redundancy,  develop 
curriculum  models  based  on  the  clusters,  and  then  design  specifications  for 
the  curriculum  models.  Finally,  the  contractor  was  required  to  provide  two 
tests  for  each  MOS  and  for  the  CST  (Hahn  &  Edwards,  1983). 

The  second  MOS  Baseline  Sicills  project  called  for  development  of  func¬ 
tionally  oriented  (job-related)  curriculums  for  two  Signal  School  MOS;  05C, 
radio  teletype  operator,  and  31 M,  multichannel  communications  equipment 
operator.  We  were  requested  to  evaluate  these  projects  while  development 
was  in  process  so  as  to  provide  guidelines  and  recommend  changes  prior  to 
production  of  final  materials. 

The  following  two  sections  describe  the  procedures  used  and  the  findings 
that  resulted  from  evaluating  these  efforts. 

Baseline  S)cills  for  94  High-Density  MOS  and  CST.  TRADOC's  contractor 
produced  a  taxonomy  of  prerequisite  competencies  needed  to  perform  job  tasks 
for  94  MOS  and  CST.  Then  TRADOC  provided  us  with  a  sample  of  job  tasks  and 
associated  competencies  for  30  of  the  94  MOS  so  that  we  could  evaluate  the 
taxonomic  structure.  Ihe  sample  included  combat  arms,  combat  support,  and 
combat  service  support  jobs.  Selected  staff  members  were  provided  with  major 
and  minor  category  titles  and  taxonomic  terms  and  were  requested  to  develop 
their  own  taxonomies  by  sorting  prerequisite  competencies  into  categories 
judged  to  be  appropriate.  These  results  were  then  compared  to  the  taxonomy 
constructed  by  TRADOC's  contractor  to  determine  the  extent  to  which  both 
groups  of  analysts  agreed  about  the  structure. 

Results  showed  complete  agreement  between  the  two  groups  of  analysts  on 
22%  of  the  competencies  sorted  into  categories  and  complete  disagreement  on 
11%  of  the  competencies.  The  taxonomic  categories  were  not  found  to  be 
mutually  exclusive  and  they  did  not  all  reflect  the  same  level  of  organiza¬ 
tion  (i.e.,  there  were  generic  categories  and  mixed  categories,  content 
categories  and  process  categories).  There  was  substantial  variability 
in  the  levels  of  specificity  of  categories  and  subcategories.  We  found 
redundant  major  categories  and  misplaced  subcategories.  The  correlation 


coefficient  between  the  two  groups  of  analysts  was  .68.  The  contractor  pro¬ 
vided  TRADOC  with  a  detailed  analysis  to  guide  ongoing  development  (Hahn  & 
Edwards,  1983). 

We  were  also  requested  to  evaluate  a  sample  of  the  design  specifications 
produced  by  TRADCXl's  contractor  to  provide  a  basis  for  curriculum  development 
based  on  the  taxonomy  of  prerequisite  competencies.  Our  recommendations  in¬ 
cluded  the  integration  of  military  language  and  materials  into  the  lessons  to 
be  developed  (including  indicators  where  mastery  of  lower  level  skills  was 
essential  to  performance  of  higher  level  skills),  the  need  for  specific  and 
clearly  written  learning  guidelines,  and  the  need  to  correct  confusing  techni¬ 
cal  language,  inconsistent  terminology,  and  writing  errors  (Stoddart  &  Hahn, 
1983a).  Guidelines  were  provided  to  TRADOC  to  assist  in  the  development  of 
the  MOS  Baseline  Skills  design  specifications. 

The  final  step  in  this  effort  will  be  to  evaluate  the  validation  of  the 
locator  and  diagnostic  tests  produced  for  each  MOS  and  CST. 

The  next  section  describes  evaluation  of  BSEP  II  programs  and  includes  a 
Needs  Assessment  for  a  program  under  development,  the  Job  Skills  Education  Pro¬ 
gram  (JSEP),  and  evaluation  of  a  pilot  program  to  teach  learning  strategies. 

BSEP  II  PROGRAMS 

Basic  Literacy  Programs 

These  programs,  offered  at  unit  sites,  provide  soldiers  whose  screening 
tests  resiilt  in  reading  levels  below  grade  nine  with  remedial  literacy  train¬ 
ing.  In  addition  to  the  methods  already  mentioned  by  which  soldiers  enroll  in 
i  '';":? — i  referral,  voluntary  requests,  and  GT  scores  below  criterion  — 

:t  also,  enrolls  soldiers  who  fail  to  meet  crlt'orion  SQT  scores. 

I'hr.  came  data  files  analyzed  for  P.oEo  r  for  n  with 

f  -'  same  overall  results:  higher  average  pay  gral  '  ,  .ewer  attrition  rates, 
gains  in  grade  levels,  and  no  significant  effects  of;  ilER  or  SQT  scores  when 
participants  were  compared  to  eligible  nonparticipants.  A  notable  difference 
between  BSEP  I  and  II  graduates  was  found  in  grade-level  gains.  The  average 
grade-level  gains  for  BSEP  II  graduates  were  about  twice  the  magnitude  of 
gains  achieved  by  BSEP  I  graduates.  In  addition,  BSEP  II  graduates  showed 
an  average  gain  in  GT  scores  of  1 7  points. 

We  traveled  to  Germany  and  Panama  to  carry  out  a  nuiiber  o1  program  eval- 
wii'ch  was  ackie:  Cc:  n  n  :  ..  .ncr.mmiKi  ioried  cffictrc 

■  •  •  zy.eriences  with  and  opiinicrs  o'  II  and  its  ernluc  t‘  ' 

■  -  ■  -  .n,  •3)  . 

Intf!r vi  ews  v«  th  Commanders  and  Noncommissioned  Off i c e r s 

The  general  tendency  was  that  Commanders  permitted  soldiers  to  enroll 
in  I  IIP  II  as  a  reward  for  good  performance.  Most  soldiers  enrolled  to 
improve  their  GT  scores.  Their  supervisors  reported  that  the  greatest  need 
was  for  BSEP  to  train  soldiers  in  reading,  then  in  writing,  and  listening 


skills.  However,  supervisors  felt  that  the  greatest  benefits  resulting  In 
BSEP  training  were  In  improved  self-esteem,  attitudes,  and  motivation.  Al¬ 
though  job  performance  benefits  were.  In  supervisors'  opinions,  secondary, 
these  supervisors  admitted  that  their  soldiers  improved  in  terms  of  motivatlcxi 
to  perform  their  jobs,  that  they  required  less  supervision,  and  that  more 
progreun  graduates  performed  in  the  top  third  of  all  unit  soldiers  than  they 
did  before  attending  BSEP  classes. 

Supervisors  had  mixed  opinions  about  the  way  that  unit  activities  are 
disrupted  when  soldiers  attend  classes.  Although  they  found  these  disruptions 
inconvenient,  they  still  thought  soldiers  gained  enough  from  BSEP  to  make  it 
worthwhile.  Overall,  the  Commanders  and  NCOS  were  of  the  opinion  that  BSEP  II 
training  had  positive  effects  on  soldiers'  performance  (Stoddart  &  Hahn, 

1983) . 


Job  Skills  Education  Program 

A  standardized,  functionalized,  computer-based  BSEP  II  program  is  under 
development  and  is  e^qwcted  to  be  fielded  in  1986.  In  order  for  sponsors  and 
developers  to  be  guided  by  concerns  and  requirements  of  field  personnel,  we 
conducted  a  JSEP  Needs  Assessment.  Ihe  Needs  Assessment  not  only  revealed 
field  personnel's  feelings  about  the  forthcoming  program,  but  also  resulted 
in  a  substantial  body  of  information  about  existing  literacy  programs.  Al¬ 
though  these  results  are  summarized  also,  they  are  provided  in  somewhat 
greater  length  in  order  to  give  a  more  comprehensive  picture  of  the  way  BSEP 
II  functions  in  the  field.  Those  questioned  and  intervle«fed  included  sol¬ 
diers  participating  in  BSEP  II,  teachers,  first- line  supervisors  and  Command¬ 
ers  in  the  Development  and  Readiness  Command,  Forces  Command,  Health  Sciences 
Command,  Training  and  Doctrine  Command,  Western  Command,  Germany,  and  Korea 
(Stoddart  &  Hahn,  1985). 

Participating  Soldiers.  Most  soldiers  enrolled  in  the  BSEP  II  programs 
that  we  covered  had  high  school  diplomas  or  equivalent  certificates.  The 
programs  in  which  they  enrolled  usually  used  self-paced,  modular  curricula 
on  %fhlch  soldiers  worked  independently.  The  soldiers  expressed  willingness 
to  have  course  materials  presented  by  computer  (JSEP  presentation  also  is 
eiqpected  to  be  self-paced,  modular,  and  worked  on  independently).  Partici- 
pauits  ei^ressed  favorable  opinions  of  teachers,  satisfaction  with  existing 
course  content,  and  comfort  with  the  difficulty  level  of  the  course  mate¬ 
rials.  Generally,  soldiers  claimed  that  BSEP  II  was  worth  their  time  and 
effort  and  also  reported  a  preference  for  morning,  on-duty  classes  that  start 
at  the  beginning  of  their  duty  assignments  (in  contract  to  Commanders  and 
NCOS  who  would  prefer  that  classes  be  scheduled  prior  to  duty  assignments). 
Most  soldiers  reported  having  enough  out-of-class  time  to  study  and  some 
felt  that  the  programs  should  be  of  Icmger  duration. 

Some  differences  were  found  between  basic  skills  needs  perceived  by 
Ccsmanders  and  NCOS  and  those  of  soldiers  enrolled  in  BSEP  ll.  Soldiers 
had  somewhat  more  confidence  in  their  reading,  writing,  and  listening  skills 
than  did  their  supervisors.  In  reading,  soldiers  reported  that  they  mainly 
needed  to  learn  rules  about  how  thin^  work  and  how  to  order  job  steps. 
Writing  and  listening  are  skills  important  to  job  performance,  soldiers 
acknowledged,  but  they  felt  confident  that  they  could  handle  job  demands  in 
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these  areas.  Soldiers  also  reported  that  the  most  useful  skill  they  could 
learn  to  enhance  job  performance  involved  memorizing  materials.  In  contrast. 
Commanders  and  NCOS  pointed  to  soldiers'  needs  in  basic  reading,  writing, 
and  listening,  and  emphasized  that  soldiers  need  to  learn  to  pay  attention 
to  details  and  to  complete  jobs  they  have  started.  Ttie  point  here  is  that 
some  soldiers  may  be  overconfident  of  their  abilities  and  harbor  inaccurate 
perceptlcms  of  their  basic  skills  needs. 

Another  finding  concerns  soldiers'  use  of  military  manuals.  Soldiers 
reported  infrequent  use  of  manuals,  although  the  one  they  were  most  likely 
to  use  was  the  Soldier's  Manual.  Their  greatest  problems  with  this  manual 
were  matching  pictures,  diagrams,  and  schematics  to  equipment  or  terrain, 
and  extracting  information  from  charts  and  graphs. 

Commanders  and  NCOs.  Supervisors  of  BSBP  II  soldiers  expressed  »d.lling- 
ness  to  support  the  programs  and  release  soldiers  to  attend  classes  in  spite 
of  disruptive  effects  on  training  schedules.  Supervisors  felt  that  the 
programs  were  worthwhile  because  they  improved  soldiers'  jda  performance,  but 
reacted  positively  to  the  alternative  of  scheduling  BSEP  training  prior  to 
duty  assignments  in  order  to  reduce  interference  with  work  schedules.  These 
supervisors  reported  that  the  durations  of  the  programs  are  appropriate  and 
that  classes  entailing  4  to  6  hours  a  day  seem  reasonable. 

Teachers.  Most  teachers  we  contacted  had  received  some  specialized 
training  to  teach  BSEP  il.  Most  had  held  their  teaching  jobs  for  only  1 
year  or  less,  indicating  a  high  turnover.  Their  jobs  included  making  class¬ 
room  presentations,  tutoring  individual  soldiers,  administering  and  scoring 
tests,  keeping  records,  and  developing  classroom  materials.  Some  of  the 
teachers  had  incorporated  Army-relevant  materials  into  their  classes.  Many 
teachers  also  acknowledged  their  lack  of  detailed  knowledge  of  military  sub¬ 
jects.  when  asked  about  the  test  used  to  screen  soldiers  for  eligibility 
(TABE),  teachers  reported' it  to  be  generally  satisfactory  but  lacking  finer 
diagnostic  characteristics  at  reading  grade  levels  below  five. 

Teachers  tended  to  disagree  with  Commanders  and  NCOs  about  command  sup¬ 
port  for  BSEP  ll,  and  pointed  to  repeated  instances  when  soldiers  were  with¬ 
drawn  from  classes  for  work  assignments.  In  terms  of  the  proposed  JSBP 
program,  teachers  expressed  concern  that  they  would  be  usurped  by  computers 
and  that  soldiers  would  be  deprived  of  the  personal  attention  and  encourage¬ 
ment  they  need.  Teachers  also  expressed  some  reservations  about  soldiers' 
ability  to  operate  computer  equipment. 

Perceived  Advantages  and  Disadvantages  of  JSEP.  A  summary  of  our 
assessment  reveals  that  JSEP  has  a  number  of  potential  advantages  over  the 
basic  literacy  courses  currently  provided  in  BSEP  II.  Since  it  is  to  pro¬ 
vide  job-related  curricula,  JSEP  is  expected  to  improve  job  skills  and  unit 
morale.  JSEP  curriculum  will  be  common  across  BSBP  II  sites,  its  teacher 
tr^dnlng  %rill  be  uniform,  and  because  it  will  be  computer  based,  it  could 
reduce  the  Impact  of  high  teacher  turnover,  reduce  the  need  for  teachers  to 
plan  instruction,  and  provide  ease  of  record  keeping,  curriculum  updating, 
test  scoring,  and  instruction  at  remote  sites. 

Some  reservations  about  the  new  program  include  possibilities  of  equip¬ 
ment  breakdowns,  delayed  repairs,  power  failures,  fatigue  or  boredom  as  a 
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result  of  long  periods  of  work  at  computer  consoles,  reduction  in  interper¬ 
sonal  exchanges  between  teachers  and  students,  teachers'  resistance  to  being 
replaced  by  computers,  and  a  need  for  teachers  to  become  familiar  with  a 
broad  variety  of  MOS  tasks  (Stoddart  &  Hahn,  1985). 

JSEP  Lesson  Specifications.  We  were  provided  with  approximately  10% 
of  the  estimated  total  number  of  lesson  specifications  to  be  prepared  for 
JSEP.  Our  purpose  in  evaluating  them  during  the  process  of  development  was 
to  provide  guidelines  to  developers  early  so  that  any  needed  redirection  of 
effort  could  Jae  made  prior  to  project  completion.  We  distributed  the  speci¬ 
fications  to  two  staff  members  with  expertise  in  curriculum  development  and 
requested  that  they  evaluate  them  in  terms  of  their  suitability  for  produc¬ 
tion  of  curricula  for  the  JSEP  program.  Then  we  consolidated  their  comments 
and  recommendations. 

In  general  terms,  the  specifications  were  of  a  sufficiently  high  qual¬ 
ity  to  provide  curriculum  designers  with  ample  guidance  for  producing  an 
organized  curriculum  that  would  meet  learning  objectives.  However,  there 
were  a  few  points  that  merited  consideration. 

First,  developers  should  keep  in  mind  that  a  totally  computer-based 
program  is  a  radical  departure  from  existing  presentation  modes  and  could 
require  extensive  orientation  and  adaptation  on  the  part  of  the  users. 

In  this  same  context,  the  specifications  made  no  provision  for  teacher- 
student  interaction. 

Second,  the  specifications  would  be  more  helpful  to  curriculum  develop¬ 
ers  if  they  included  more  detailed  guidance  about  the  intended  complexity  of 
instruction  to  be  given  to  each  of  the  prerequisite  competencies  that  will 
be  taught.  (These  prerequisite  competencies  are  those  developed  for  94 
high-density  MOS  and  CST  discussed  above  under  MOS  Baseline  Skills.)  In 
addition,  it  is  important  to  be  aware  that  some  of  the  competencies  were 
originally  assigned  to  the  wrong  job  tasks.  These  assignments  need  to  be 
reviewed,  and  it  was  also  recommended  that  relationships  among  competency 
categories  and  job  tasks,  test  items,  and  learning  objectives  be  defined 
more  explicitly.  In  some  cases,  prerequisite  competencies  could  be  combined 
(e.g.,  four  competencies  associated  with  outlining)  so  that  a  single  lesson 
could  be  designed  that  incorporated  all  of  them. 

Finally,  we  recommended  that  some  presentation  features  be  reconsidered 
on  the  basis  of  their  suitability  for  an  adult  population  (e.g.,  interjected 
cartoon  figures).  Some  of  the  rewards  and  incentives  included  in  the  speci¬ 
fications  also  seemed  somevdiat  juvenile  and  needed  to  be  reevaluated  (C.  P. 
Hahn,  personal  communication,  October  1983). 


Learning  Strategies 

In  1981  a  pilot  program  intended  to  embed  learning  strategies  in  BSEP  II 
curriculum  content  was  conducted  at  Fort  Knox.  The  objective  of  the  pilot 
was  to  determine  if  training  soldiers  in  cognitive  skills  (i.e.,  strategies 
that  they  could  use  to  organize,  interpret,  and  remember  material  to  be 
learned)  would  enhance  their  acquisition  of  basic  skills.  The  particular 
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program  selected  for  tryout  was  an  adaptation  of  Feuerstein's  Instrumental 
Enrichment  Program  (IE)  (Russ-Eft  &  McLaughlin,  1983). 

The  IE  program  consists  of  14  instruments  (exercise  boolclets)  that  can 
be  used  to  teach  basic  cognitive  skills  such  as  problem  definition,  analytical 
thinking,  systematic  searching,  perceptual  precision,  and  learning  processes. 
Hie  instruments  were  designed  to  be  used  with  culturally  deprived  adolescents 
and  led  to  increased  scores  on  standard  tests.  Hie  objective  of  the  program 
is  to  change  passive,  dependent  cognitive  responses  to  autonomous,  independent 
responses . 

The  Army  tryout  did  not  accomplish  the  desired  goals.  That  is,  soldiers 
did  not  demonstrate  significantly  improved  levels  of  either  cognitive  skills 
or  posttest  scores  on  basic  literacy  tests.  Hie  tryout  pinpointed  areas  of 
possible  difficulties  in  administering  the  IE  program  so  that  future  adapta¬ 
tions  could  avoid  such  pitfalls.  Our  evaluation  provided  this  information. 

The  adapted  version  of  the  IE  program  reduced  2  years  of  instruction  to  6 
weeks  (soldiers  with  higher  GT  scores)  and  12  weeks  (soldiers  with  lower  GT 
scores),  and  included  only  a  sample  of  the  instruments  used  in  the  full-scale 
IE  program.  Also,  the  full-scale  program  was  developed  for  culturally  de¬ 
prived  adolescents,  not  adult  military  personnel.  Because  the  2-year  progreun 
administered  to  adolescents  was  successful,  attenuating  the  program  and  using 
it  with  adults  may  .have  diluted  its  effectiveness.  Another  possible  important 
difference  is  that  Feuerstein's  posttest  data  were  gathered  5  years  after  the 
instruction  was  given.  The  soldiers  at  Fort  Knox  were  tested  for  program 
effects  immediately  after  the  coursework  was  completed. 

Another  factor  that  may  have  contributed  to  the  lack  of  substantive 
effects  was  inadequate  teacher  training.  Teachers  never  fully  appreciated 
the  importance  of  integrating  IE  materials  into  coursework  and  spending  the 
full  amount  of  time  called  for  on  each  instrument.  The  teachers  expressed 
ongoing  concern  that  they  were  neglecting  basic  literacy  subjects  which  sol¬ 
diers  needed  badly.  Furthermore,  instructors  never  really  mastered  the 
necessary  skills  for  relating  individual  learning  strategies  to  real  world 
situations.  All  of  this  implies  the  teachers'  incomplete  understanding  of 
the  nature  of  these  learning  strategies. 

An  additional  circumstance  that  affected  the  tryout  was  termination  of 
the  teachers'  contracts  part  way  through  implementation  of  IE.  Ohis  event 
constituted  a  clear  morale  problem  so  that,  by  the  end  of  the  cycle,  some 
teachers  were  not  using  any  IE  materials  at  all  in  their  classes. 

It  is  important  to  note  that  the  soldiers  who  were  provided  with  IE 
instruction  demonstrated  gains  in  test  scores  just  as  great  as  those  of 
soldiers  in  classes  in  which  IE  was  not  included.  Furthermore,  it  has  not 
been  determined  whether  strategies  taught  during  the  Fort  Knox  tryout  will 
have  long-term  effects  on  these  soldiers'  learning  abilities.  Given  all  of 
the  above  circumstances,  it  seems  reasonable  to  conclude  that  the  IE  did  not 
receive  an  adequate  tryout. 

The  final  section  summarizing  our  evaluation  activities  covers  ESL 
programs  and  includes  a  discussion  of  the  ECLT,  11B  Language  Skills,  baseline 
programs,  pilot  programs,  and  the  new  functionalized  program. 
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ENGLISH-AS-A-SECOND-LANGUAGE  PROGRAMS 

The  Army  provides  ESL  training  both  at  training  bases  as  part  of  BSEP  1 
and  at  unit  sites  as  part  of  BSEP  II.  Soldiers  are  screened  for  eligibility 
by  means  of  the  English  Comprehension  Level  Test  and  those  with  scores  lower 
than  70  (range  0  to  100)  are  considered  in  need  of  English-language  instruc¬ 
tion.  Most  enrolled  soldiers  are  members  of  the  Active  Army.  Enlisted  Re¬ 
serve  and  National  Guard  members  tend  to  reject  the  offer  of  ESL  training. 

Although  soldiers  needing  English-language  training  constitute  only  a 
small  part  of  the  enlisted  force  (approximately  5%),  they  receive  much  at¬ 
tention.  One  reason  for  this  special  focus  is  that  limited-English-speaking 
(LES)  soldiers,  most  of  whom  are  insular  Puerto  Rican,  tend  to  be  a  well- 
educated  and  highly  motivated  group  who,  equipped  with  adequate  English- 
language  skills,  become  an  asset  to  the  military.  An  additional  reason  is 
that  the  Army  is  confronting  a  shrinking  recruitment  pool  during  the  1980s 
and  anticipates  admitting  a  growing  proportion  of  Hispanic  recruits.  As 
a  result,  the  Army  has  supported  pilot  ESL  programs  as  vrell  as  development  of 
a  functionalized  curriculum  to  enable  LES  soldiers  to  succeed  in  training 
and  job  performance. 

English  Comprehension  Level  Teat 

The  ECLT  has  been  used  as  the  index  of  English-language  proficiency  at 
DLI  for  more  than  20  years.  However,  since  the  test  requires  only  reading 
and  listening,  some  questions  arose  about  its  value  when  used  to  test  sol¬ 
diers  with  marked  deficiencies  in  spoken  English.  As  a  result,  we  developed 
an  individually  administered  test  called  the  Oral  Proficiency  Test  that  was 
based  on  a  test  provided  by  DLI  and  that  required  both  speaking  (production 
including  vocabulary,  grammar,  fluency,  and  pronunciation)  and  understanding 
spoken  English  (comprehension).  He  used  this  test  with  86  participants  in 
a  3-month  pilot  ESL  program  provided  by  DLI  and  in  6-week  BSEP  I  programs. 

Correlations  between  Oral  Proficiency  scores  and  ECLT  scores  were  high 
(.70  to  .89)  with  the  exception  of  one  subtest — pronunciation.  Soldiers 
who  had  received  only  6  weeks  of  ESL  training  showed  no  gains  in  scores 
on  this  subtest.  A  high  correlation  was  also  found  between  ECLT  scores 
and  ratings  by  BCT  and  AIT  instructors  of  soldiers'  English-language  pro¬ 
ficiency  as  well  as  with  attrition  in  the  training  base  (Harman,  Oxford- 
Carpenter,  &  Redish,  1983). 

MOS  11B  Language  Skills 

During  the  initial  stage  of  our  evaluation  activities,  we  conducted 
a  needs  assessment  involving  limited  English-speaking  and  native  English- 
speaking  soldiers  %dio  were  participating  in  AIT  for  MOS  11B  at  Fort  Banning. 
He  developed  a  test  using  Job  Language  Performance  Requirements  for  MOS  11B 
from  an  analysis  produced  by  DLI.  The  outcome  was  that,  on  the  average,  LES 
soldiers  missed  more  than  25%  of  the  test  items  and  those  who  had  ECLT  scores 
below  70  missed  approximately  50%.  ‘rtiese  results  contrast  with  those  of  na¬ 
tive  English-speaking  soldiers  who  missed  about  5%  of  the  test  items.  A 
small  number  of  soldiers  participated  in  this  assessment  so  that  the  findings 
must  be  considered  indicative  rather  than  definitive  (Holland,  1982). 
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Six-week  programs  using  the  American  Language  Course  developed  and  used 
by  DLI  were  part  of  BSEP  at  the  beginning  of  our  evaluation  efforts.  We 
gathered  data  on  these  programs  not  only  to  assess  program  effectiveness,  but 
also  to  serve  as  baseline  information  for  comparing  pilot  (3-month  and  6-month) 
and  revised  (functionalized)  programs.  We  visited  seven  TRADOC  (BSEP  I)  sites 
during  1981  and  1982  to  administer  questionnaires  and  interview  Education  Ser¬ 
vices  Offices'  staff  members,  teachers,  and  participants  and  their  supervisors. 

Most  programs  used  the  American  Language  Course  as  the  curriculum  and 
supplemented  it  with  military  information  (e.g.,  Training  Extension  Course 
tapes,  BCT  charts.  Soldier's  and  Field  Manuals,  etc.).  Classes  tended  to 
include  soldiers  trith  wide-ranging  language  deficiencies.  This  was  a  result 
of  the  open-entry  system  that  permitted  soldiers  to  join  classes  in  progress 
as  they  reported  to  the  post  and  were  screened  for  eligibility.  In  addition, 
enrollments  fluctuated  and,  when  enrollments  were  low,  it  was  not  cost  effec¬ 
tive  to  separate  soldiers  on  the  basis  of  language-deficiency  levels . 

Instruction  for  most  programs  was  provided  by  contractors.  In  some  in¬ 
stances,  ESOs  hired  teachers  by  means  of  nonpersonal  services  contracts.  In 
both  cases,  lowest  bidding  contractors  or  lowest  bidding  teachers  had  to  be 
selected.  This  turbulence  could  occur  on  a  yearly  basis.  The  result  was 
high  teacher  turnover  and  low  morale  as  salaries  declined  from  year  to  year. 

It  also  required  repeated  warmup  periods  and  reduced  services  as  new  con¬ 
tractors  took  over  direction  of  the  programs  from  year  to  year. 

Soldiers  enrolled  in  the  programs  had  widely  distributed  ECLT  scores, 
although  more  scores  were  distributed  in  the  low  (0-29)  and  middle  (30-49) 
ranges  than  in  the  high  range  (50-69).  Eligible  soldiers  who  did  not  enroll 
in  BSEP  for  English-language  training  tended  to  have  ECLT  scores  at  the  high 
end  of  the  distribution.  This  was,  in  part,  because  those  who  did  not  enroll 
when  the  initial  opportunity  was  available  and  who  had  pronounced  deficiencies 
tended  to  enroll  after  they  demonstrated  communications  problems.  All  of  the 
soldiers  interviewed  indicated  prior  knowledge  of  English — most  frequently  in 
school  settings  in  which  they  were  taught  to  read  and  write  English  but  had 
little  experience  with  English  conversation  (Holland,  Rosenbaum,  Stoddart, 
Redish,  Harman,  &  Oxford-Carpenter,  1984). 

Soldiers  tended  to  feel  positively  about  the  ESL  programs  and  their 
teachers.  At  posts  where  participants  vere  billeted  with  English-speaking 
soldiers,  they  had  opportunities  to  practice  what  they  learned  in  classes. 
However,  many  participants  were  assigned  to  ESL  companies,  and  most  ESL 
soldiers  we  interviewed  said  that  the  classroom  instruction  included  too 
little  English  conversation.  Many  also  pointed  to  a  need  for  more  time  to 
study  outside  of  classes.  About  half  of  the  participants  felt  that  the 
6-week  program  was  too  brief. 

Soldiers  who  graduated  from  the  program  improved  on  ECLT  scores  by  an 
average  of  two  points  per  week.  Clearly,  those  who  entered  the  program  with 
scores  at  the  lower  end  of  the  distribution  failed  to  meet  the  criterion 
score  of  70  during  the  6-week  period.  We  found  positive  correlations  between 
ECLT  exit  scores  and  AIT  success,  attrition,  and  recycling,  and  these  effects 
%iere  more  pronounced  for  soldiers  whose  exit  scores  were  above  50. 


A  6-month  ESL  program  was  conducted  at  DLI  during  the  fall  of  1980  and 
the  winter  of  1981.  'nie  program  was  ending  at  the  time  that  our  evaluation 
activities  began.  However,  we  visited  DLI  to  obtain  data  gathered  during 
the  6-month  period  and  to  interview  teachers  and  program  administrators. 

Of  particular  interest  was  a  group  of  200  soldiers  selected  as  eligible  for 
ESL  participation  but  who  were  not  enrolled  in  the  program.  These  soldiers 
served  as  a  control  group  for  measuring  program  effects.  B(}th  the  enrolled 
and  eligible  nonenrolled  soldiers  volunteered  either  to  participate  in  the 
6-month  prograun  or  to  go  directly  to  BCT. 

During  the  sumnmer  and  fall  of  1981,  a  3-month  ESL  program  was  provided 
at  DLI.  We  gathered  interview,  questionnaire,  and  testing  data  on  this  pilot 
program  also. 

Language  acquisition  in  both  pilot  programs  tended  to  be  the  same  orderly, 
predictable  phenomenon  that  was  demonstrated  in  6-wee)c  ESL  programs.  Partici¬ 
pants  tended  to  gain  two  ECLT  points  per  week.  In  contrast,  the  group  of  sol¬ 
diers  who  were  screened  as  eligible  but  who  went  directly  to  basic  training 
sites  showed  average  gains  of  only  one-half  ECLT  point  per  week.  These  sol¬ 
diers  were  retested  between  6  and  9  months  following  initial  screening  and 
some  of  them  had  been  enrolled  in  6-week  ESL  programs  at  training  bases.  In 
addition,  attrition  was  significantly  greater  for  control  group  soldiers  than 
for  pilot  program  graduates.  Another  finding  common  to  the  pilot  programs 
and  standard  6-week  programs  was  that  soldiers  felt  that  instruction  should 
include  more  English  conversation  and  less  reading. 

We  followed  as  many  program  graduates  as  we  could  into  BCT  and  AIT.  We 
are  still  analyzing  data  on  long-range  effects  and  will  report  our  results 
later  this  year.  Early  indications  are  that  there  is  a  high  correspondence 
between  soldiers'  final  ECLT  scores  and  ratings  of  their  English  proficiency 
by  drill  sergeants  and  AIT  instructors.  There  was  a  distinction  between  sol¬ 
diers  who  scored  below  ECLT  50  and  those  who  had  exit  scores  of  50  or  more 
in  terms  of  their  supervisors'  ratings.  Program  graduates  reported  that 
their  greatest  language  deficiency  continued  to  be  speaking  skills.  However, 
most  said  that  the  programs  at  DLI  had  equipped  them  with  sufficient  English- 
language  skills  to  complete  BCT  and  AIT  (Stoddart,  1982). 

Functionalized  ESL  Program 

In  July  1982  TRADOC  implemented  a  job-related  ESL  curriculum  at  eight 
sites.  DLI  personnel  developed  the  curriculum  which  was  designed  to  provide 
soldiers  with  6  weeks  of  training  in  English-language  skills  as  well  as  with  a 
vocabulary  that  would  facilitate  their  progress  through  basic  training.  The 
curriculum  was  developed  for  soldiers  with  entry  ECLT  scores  from  50  to  69. 

Our  evaluation  of  this  program  started  in  August  1983  and  is  continuing  at 
this  writing.  The  findings  detailed  below  represent  information  gathered  to 
date.  Whether  any  changes  in  present  trends  will  be  documented  by  the  end 
of  the  evaluation  effort  remains  to  be  seen  (Rosenbaum  &  Stoddart,  1985). 


Although  the  functionalized  ESL  program  curriculum  was  in  use  at  all  the 
sites  we  visited,  its  presentation  varied  as  a  function  of  supplementary  ma¬ 
terials  added  by  teachers  (Training  Extension  Course  tapes,  SMART  books,  etc.) 


and  fluctuating  enrollment  patterns.  At  a  few  posts,  program  directors  decided 
to  adhere  strictly  to  curriculum  materials  and  made  no  attempt  to  introduce 
ancillary  materials. 

The  distribution  of  ECLT  entry  scores  for  soldiers  participating  in 
the  job-related  program  has  shifted  markedly  from  a  majority  with  low-  and 
mid-level  scores  to  a  majority  with  mid-  and  high-level  scores.  Nonetheless, 
more  than  50%  of  the  entry  scores  are  still  below  50,  and  this  means  that 
these  soldiers  are  not  part  of  the  target  population  for  whom  the  curriculum 
was  developed.  Participants  with  lower  entry  scores  spend  the  first  2  weeks 
of  instruction  on  basic  English  grammar  and  vocabulary  before  advancing  to 
the  functionalized  curriculum.  Scores  on  the  end-of -course  test  developed 
to  accompany  the  curriculum  reflect  the  need  for  extra  instructional  attention 
on  the  part  of  lower  scoring  soldiers.  Participants  in  the  target  population 
are  more  likely  to  pass  this  test  than  are  those  who  enter  the  program  with 
ECLT  scores  below  50  (2/3  vs.  1/3).  Overall  ECLT  gains,  however,  averaged 
just  about  two  points  per  week  just  as  they  did  in  other  programs.  This 
similarity  must  be  viewed  in  the  context  of  a  military-based  curriculum. 

The  ECLT  includes  no  military  content,  so  that  comparable  achievements  on 
the  part  of  functionalized  program  graduates  represent  a  somewhat  higher 
level  of  accomplishment. 

In  terms  of  magnitude  of  ECLT  gains,  the  data  are  a  bit  different  from 
those  gathered  on  other  ESL  programs.  The  nontarget  population  (ECLT  scores 
below  50)  showed  substantially  greater  gains  than  did  participants  whose  entry 
scores  were  50  or  higher.  This  may  be  because  these  soldiers  spend  the  ini¬ 
tial  period  of  instruction  studying  fundamental  language  skills. 

It  is  important  to  note  that  the  change  introduced  in  the  ESL  programs 
consisted  only  of  a  new  curriculum.  All  the  other  features  of  the  standard 
ESL  programs  that  may  have  limited  their  effectiveness— the  low-bid  system, 
the  teachers  without  prior  ESL  experience,  the  classes  incorporating  wide- 
ranging  skill  levels,  the  soldiers  assigned  to  ESL  companies,  and  the  lack 
of  emphasis  on  English  conversation — must  be  assumed  to  have  the  same  impact 
on  the  job-related  program.  However,  participating  soldiers  expressed  very 
positive  opinions  about  the  progreun  and  the  teachers  and  reported  that  the 
progr£un  provided  them  with  enough  English-language  skills  to  succeed  in 
training  (Rosenbaum  &  Stoddart,  1985). 


SUMMARY 

A  general  overview  of  BSEP  reveals  that  it  improves  soldiers'  basic 
skills  and  that  the  improvement  is  greater  for  soldiers  whose  entry  scores 
are  very  low.  Those  who  benefit  most  directly  from  BSEP,  participating  sol¬ 
diers  and  their  supervisors,  acknowledge  that  the  program  is  worthwhile  and 
point  to  enhanced  motivation  and  self-esteem.  However,  it  is  important  to 
note  that  a  substantial  number  of  soldiers  leave  the  program  without  meeting 
criterion  standards.  These  soldiers  enter  basic  training  or  return  to  their 
unit  assignments  regardless  of  performance  in  BSEP,  and  this  circumstance  can 
lead  to  adverse  criticism  of  program  achievements  as  well  as  to  dissatis¬ 
faction  on  the  part  of  Commanders. 


An  equally  important  point  is  that  the  Army  is  continually  trying  to 
update  and  improve  BSEP  to  ensure  that  it  more  closely  meets  program  goals. 
Hie  variety  of  pilot  and  revised  programs  discussed  in  this  report — e.g.. 
Instrumental  Enrichment,  Job  Skills  Education,  extended-duration  ESL  and 
functionalized  ESL  programs — are  just  a  few  examples  of  this  ongoing  con¬ 
cern.  In  line  with  this  interest,  the  final  task  of  our  evaluation  will  be 
to  develop  and  test  a  Quality  Control  System  that  will  provide  continuous 
information  about  the  status  and  effectiveness  of  all  basic  skills  programs 
throughout  the  Army. 
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